New Algorithms for Heavy Hitters in Data Streams (Invited Talk)

An old and fundamental problem in databases and data streams is that of finding the heavy hitters, also known as the top-k, most popular items, frequent items, elephants, or iceberg queries. There are several variants of this problem, which quantify what it means for an item to be frequent, including what are known as the l1-heavy hitters and l2-heavy hitters. There are a number of algorithmic solutions for these problems, starting with the work of Misra and Gries, as well as the CountMin and CountSketch data structures, among others. In this survey paper, accompanying an ICDT invited talk, we cover several recent results developed in this area, which improve upon the classical solutions to these problems. In particular, with coauthors we develop new algorithms for finding l1-heavy hitters and l2-heavy hitters, with significantly less memory required than what was known, and which are optimal in a number of parameter regimes. 1 The Heavy Hitters Problem A well-studied problem in databases and data streams is that of finding the heavy hitters, also known as the top-k, most popular items, frequent items, elephants, or iceberg quries. These can be used for flow identification at IP routers [21], in association rules and frequent itemsets [1, 25, 26, 44, 47], and for iceberg queries and iceberg datacubes [7, 22, 24]. We refer the reader to the survey [18], which presents an overview of known algorithms for this problem, from both theoretical and practical standpoints. There are various different flavors of guarantees for the heavy hitters problem. We start with what is known as the l1-guarantee: Definition 1 (l1-(ǫ, φ)-Heavy Hitters Problem) In the (ǫ, φ)-Heavy Hitters Problem, we are given parameters 0 < ǫ < φ < 1, as well as a stream a1, . . . , am of items aj ∈ {1, 2, . . . , n}. Let fi denote the number of occurrences of item i, i.e., its frequency. The algorithm should make one pass over the stream and at the end of the stream output a set S ⊆ {1, 2, . . . , n} for which if fi ≥ φm, then i ∈ S, while if fi ≤ (φ − ǫ)m, then i / ∈ S. Further, for each item i ∈ S, the algorithm should output an estimate f̃i of the frequency fi which satisfies |fi − f̃i| ≤ ǫm. We are interested in algorithms which use as little space (i.e., memory) in bits as possible to solve the l1-(ǫ, φ)-Heavy Hitters Problem. We allow the algorithm to be randomized and to succeed with probability at least 1 − δ, for 0 < δ < 1. We do not make any assumption on the ∗A preliminary version of this paper appeared as an invited paper in ICDT 2016.